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I. Introduction to the Local Systemic Change Initiative 



In the spring and summer of 1995, the National Science Foundation (NSF) funded the first 
cohort of eight projects in a new initiative, the Local Systemic Change through Teacher 
Enhancement (ESC) program. Eighteen additional projects were funded in 1996, 20 in 1997, 12 
in 1998, 13 in 1999, 9 in 2000, 7 in 2001, and 1 in 2002 for a total of 88 projects in Cohorts 1-8. 
No additional ESC projects were funded since 2002. 

The goal of the ESC program is to improve the teaching of science, mathematics, and technology 
by focusing on the professional development of teachers within whole schools or school districts. 
Each targeted teacher is to participate in a minimum of 130 hours of professional development 
over the course of the project.^ In addition to its focus on involving all teachers in a jurisdiction, 
the ESC initiative is distinguished from previous teacher enhancement efforts by its emphasis on 
preparing teachers to implement designated exemplary mathematics and science instructional 
materials in their classrooms. 

ESC projects are expected to align policy and practice within the targeted district(s) and to 
include: 

• A shared comprehensive vision of science, mathematics, and technology education; 

• Active partnerships and commitments among stakeholders; 

• A detailed self-study that provides a realistic assessment of the system’s strengths and 
needs; 

• Strategic planning that incorporates mechanisms for engaging each teacher in 
intensive professional development activities over the course of the project; and 

• A set of clearly defined, measurable outcomes for teaching, and an evaluation plan that 
provides ongoing feedback to the project. 

The ESC solicitation indicated NSE’s plan to “provide a framework for data collection 
(including a set of instruments and procedures) that will allow the Eoundation to evaluate 
individual projects, aggregate data and information across projects, and produce a cross-project 
analysis” (NSE 94-73). NSE contracted with Horizon Research, Inc. (HRI) of Chapel Hill, NC 
to design the data collection framework, provide technical assistance in its implementation, and 
analyze cross-site evaluation results. 

This section provides an overview of the ESC projects and a description of core evaluation data 
collection activities. Subsequent sections present the findings from the core evaluation activities 
conducted from September 1, 2003 through May 1, 2006. 



* Prior to 1999, the requirement for K-8 projects was 100 hours. 
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A. An Overview of LSC Projects in Cohorts 1-8 



Data provided by the Pis and questionnaires completed by the principals of targeted schools 
provide some basic information about the LSC projects included in Cohorts 1-8. 

• The LSC initiative has funded 38 K-8 science projects, 6 secondary science projects, 
18 K-8 mathematics projects, 14 secondary mathematics projects, 6 projects that 
targeted both elementary mathematics and science, 1 project that targeted both 
elementary and secondary science, and 5 projects that targeted both elementary and 
secondary mathematics. 

• Thirty-eight of the LSC projects were single-district projects; at the other end of the 
scale, 4 projects involved more than 20 districts each. 

• Sixty-six of the projects were funded as five-year projects, 14 as four-year, and 8 as 
three-year; although a number of projects have been granted no-cost extensions. 

• The 88 current and completed projects plan to involve a total of approximately 70,000 
teachers in roughly 4,000 schools in 467 districts across the United States. 

• By the completion of these projects, an estimated 2,142,000 students will receive 
instruction from LSC-treated teachers each year. 

Twenty-two LSCs were still active for at least one of the data collection years included in this 
report (see Table 1).^ Most projects targeted a single subject (e.g., K-8 mathematics); four 
projects targeted multiple subject/grade-ranges (i.e., K-12 mathematics, K-12 science, K-8 
mathematics/ science). 



Table 1 

Active LSCs, by Targeted Subject^ 





Number of Projects 


Mathematics 




K-8 


6 


6-12 


0 


K-12 


1 


Science 




K-8 


8 


6-12 


4 


K-12 


1 


Mathematics and Science 




K-8 


2 



Three LSC projects targeted middle school only; 1 is categorized as 6-12 
mathematics, 2 are categorized as 6-12 science. 



^ Three projects completed data collection in 1998, 6 in 1999, 8 in 2000, 19 in 2001, 13 in 2002, and 17 in 2003; 
these 66 projects are not included in the analyses in this report. 
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B. Schools Participating Between 2003 and 2006 



As can be seen in Figure 1, 36 percent of the schools targeted in the LSCs included in this report 
are in urban areas, 21 percent are in suburban areas, 26 percent are in rural areas, and 17 percent 
are in towns or small cities. 



Percentage of LSC Schools in Various Types of 
Communities 




Suburban 



21 % 



Figure 1 



In terms of student demographics, across all schools targeted by these LSCs, 48 percent of 
students are white, 24 percent Hispanic, 22 percent African-American, 5 percent Asian, 1 percent 
American Indian or Alaskan Native, 0.1 percent Native Hawaiian or Pacific Islander, and 0.6 
percent from another background. As can be seen in Figure 2, projects targeting K-8 
mathematics serve the largest proportion of minority students, but in each subject/grade range the 
representation of minority students is about as large as, if not larger than, the national average of 
approximately 40 percent. 

The typieal school targeted for K-8 mathematics or science reform by these LSC projects has 
478 students, 57 percent of whom qualify for free or reduced-price lunehes and 13 percent of 
whom are of limited English proficiency (LEP). The typical school targeted for 6-12 
mathematics or science reform has 633 students, 71 percent of whom are eligible for free or 
reduced-price lunches and 1 1 percent of whom are classified as EEP. 
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Figure 2 
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C. Description of Core Evaluation Data Collection and Analysis 



HRI worked with the National Science Foundation and Pis and evaluators of the early LSC 
projects on the design and implementation of a core evaluation system to allow the aggregation 
of information across projects. This section describes the data collection activities associated 
with the core evaluation. Subsequent sections of the report present results for the three core 
evaluation questions listed below, followed by a summary section. 



LSC Core Evaluation Questions 

> What is the overall quality of the LSC professional development activities? 

> What is the impact of the LSC professional development on teacher 
preparedness, attitudes, and beliefs about mathematics and science teaching 
and learning? 

> What is the impact of the LSC professional development on classroom 
practices in mathematics and science? 



Data Collection 

This report spans three data collection years, in which each project included in this report 
collected core evaluation data at least once. For most data collection activities, only the most 
recent data collected were used in these analyses.^ Data collection activities included the 
following: 

1. Observations of Professional Development Activities 

The core evaluation calls for projects to conduct 5-8 observations of professional 
development sessions each year and record their observations on standardized 
protocols. Evaluators were to consult with Pis on what professional development 
experiences were planned throughout the data collection year, and to select a sample 
that was representative of the diversity of the project’s activities. Program-wide, a 
total of 283 observations of professional development sessions were conducted in the 
period from September 1, 2003 through May 1, 2006. Data were weighted to control 
for the variable number of observations conducted per project. 



^ Data from all professional development observations conducted over these three years were included in the 
analyses described in this report. 
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2. Classroom Observations 

All 22 active projects conducted classroom observations of randomly selected 
teachers (or their backups), with the number of observations ranging from 10 to 24 
per project."^ A total of 366 classrooms were observed, roughly three-fourths of 
which were taught by teachers who had participated in at least 20 hours of LSC 
professional development. In all cases, the data were weighted to represent the total 
population of eligible teachers in the project. 

3. Teacher Questionnaires 

Twenty-one projects administered teacher questionnaires developed for the core 
evaluation to a random sample of teachers for each targeted subject; the median 
response rate among projects was 86 percent.^ A total of 4,952 teacher questionnaires 
were returned to HRI, including 2,247 from K-8 science teachers, 638 from 6-12 
science teachers, 1,999 from K-8 mathematics teachers, and 68 from 6-12 
mathematics teachers. Weights were added to the data file to reflect the probability 
of each teacher’s selection into the sample, adjusted for any non-response in that 
project. 

4. Principal Questionnaires 

Each of the 22 projects administered questionnaires to the entire population of 
principals of targeted schools. Return rates on the principal questionnaire were 
generally higher than for the teacher questionnaire; a total of 720 principal 
questionnaires were returned, with a median response rate among projects of 94 
percent. 

5. Teacher Interviews 

Each year after the baseline year, evaluators were asked to interview a random sample 
of 10 teachers who had participated in at least 20 hours of professional development 
activities in that project. A total of 231 interviews were conducted among the 22 
projects. Eighty-four percent of the interviews were conducted by phone, and 16 
percent were conducted in person. Evaluators reported the interview data by 
completing an interview summary form with both ratings and direct quotations from 
the participating teachers. Interview data from each project were weighted to reflect 
the total number of teachers who had participated in ESC professional development in 
that project. 



Projects targeting a single subject/grade-range were required to conduct 16 classroom observations in their 
baseline, second, and final years. Projects targeting multiple subjects/grade-ranges were required to conduct 12 
classroom observations per subject/grade-range in those years. All but 4 of the 22 projects included in this report 
completed the required number of observations. 

^ Although all 22 projects were responsible for submitting teacher questionnaires during the time period examined 
in this report, one project failed to do so. 
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Table 2 provides a summary of the number of projects participating in each data collection 
activity, by targeted subject.*’ 



Table 2 



Projects Participating in Each Core 
Evaluation Component, by Targeted Subject 





Number of Projects 


Science 


Mathematics 


K-8 


6-12 


K-8 


6-12 


Professional Development Observations 


11 


5 


9 


1 


Classroom Observations 


11 


5 


9 


1 


Teacher Questionnaires 


10 


5 


9 


1 


Principal Questionnaires 


11 


5 


9 


1 


Teacher Interviews 


11 


5 


9 


1 



Data Analysis 

To facilitate the reporting of large amounts of survey data, and because individual questionnaire 
items are potentially unreliable, HRI used factor analysis to identify survey questions that could 

n 

be combined into “composites.” Each composite represents an important construct related to 
one of the core evaluation questions. For example, there is a composite on the quality of ESC 
professional development, and several on teacher attitudes, preparedness, and classroom 
practice.^ 

Once the questionnaire items associated with each composite were identified, composite scores 
were created. The composites are calculated as percentages of total points possible. An 
individual teacher’ s composite score is calculated by summing his/her responses to the items in 
that composite and then dividing by the total points possible. For example, if a composite is 
based on six survey questions asked on a five-point scale of “strongly disagree” to “strongly 
agree,” that composite has 30 total possible points. If a teacher’s raw composite score on these 
six items adds to 24 points, the percentage score is 80 (computed as 24 -h 30 x 100). A project’s 
mean composite score is computed by averaging the scores of the individual teachers in that 
project. 

In the results presented in this report, teachers, schools, and projects are sometimes categorized 
by targeted subject/grade-range (K-8 science, 6-12 science, or K-8 mathematics; results for 6- 
12 mathematics are not presented separately due to the small number of participants in this 



® In projects targeting both mathematics and science, or both elementary and secondary mathematics, questionnaire, 
observation, and interview data were collected separately for each subject/grade-range combination. Thus, the sum 
of projects is greater than the total number of active projects. 

^ See “Technical Report; Analysis of the Psychometric Structure of the LSC Surveys” (12/07/98) by David B. Flora 
and A.T. Panter, L.L. Thurstone Psychometric Lab, University of North Carolina at Chapel Hill, NC for a detailed 
description of the factor analysis procedure. 

* See http;//www.horizon-research.com/LSC/news/composites/composites.pdf for definitions of the composite 
variables. 
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subject/grade-range during the period covered by this report). Analyses of the impact of the LSC 
initiative on teachers and their teaching are typically reported by extent of teacher involvement in 
LSC professional development activities.^ Differences in proportions were tested using Chi- 
square procedures. Analysis of variance and t-tests were used to test the significance of 
differences in means of continuous variables, using the Holm-Bonferroni adjustment to 
compensate for the fact that multiple comparisons were performed. Differences noted in this 
report are statistically significant at the 0.05 level. 



^ “Teacher leaders” are likely not representative of the typical teacher targeted by the LSCs and were omitted from 
these analyses. 
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II. Quality of LSC Professional Development 



A. Introduction 

For the core evaluation, project evaluators were asked to observe 5-8 professional development 
activities in each ongoing project. Evaluators and Pis were to decide jointly which activities 
would be observed, selecting sessions to represent the diversity of the project’s professional 
development offerings and to reflect the extensiveness and importance of the various kinds of 
activities. A total of 283 professional development sessions was observed during the period 
covered by this report. 

This section of the report presents a summary of data collected from observations of individual 
sessions across the 22 LSC projects active during the data collection period, including both 
descriptive information about the observed sessions and evaluators’ assessments of their quality. 
The section concludes with teacher and evaluator judgments of the overall quality of the LSC 
professional development programs. 



B. Description of LSC Professional Development Sessions 

Evaluators documented a number of features of each professional development session, 
providing information about targeted participants, presenters/facilitators, purposes and content 
focus, and the major types of activities that characterized the sessions. 

Participants 

The majority of professional development sessions observed for the LSC core evaluation 
included between 1 1 and 50 participants. Of sessions targeting teachers, 10 percent exclusively 
targeted teacher leaders, 85 percent targeted only regular teachers, and 5 percent targeted both 
lead and regular teachers. A total of 6 percent of the sessions included principals or other 
administrators. 

Presenters/F acilitators 

LSC professional development involved presenters/facilitators from a variety of settings. 
Seventy-two percent of the observed sessions included one or more district personnel as 
presenters or facilitators, while only 17 percent of the sessions included university faculty as 
presenters or facilitators. (See Ligure 3.) Across all of the observed sessions, 71 percent of the 
presenters/facilitators were female and 29 percent were male. And as can be seen in Ligure 4, 87 
percent of the presenters/facilitators were white and 13 percent were members of other 
race/ethnic groups. 



In addition to the core evaluation data collection, evaluators sometimes observed additional professional 
development activities without completing core evaluation protocols, and interviewed teachers about their 
professional development experiences, using project-specific protocols. All of the available data were to be used in 
making the summary judgments. 
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LSC Professional Development Sessions 
Including Various Types of Presenters/Facilitators 




Race/Ethnicity of Presenters/Facilitators 




Figure 4 
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Purposes of the Professional Development Sessions 

Evaluators were asked to indicate the primary intended purposes of each observed session based 
on information provided by the session facilitators. The vast majority of the observed sessions 
dealt with classroom practice, either pedagogy or the implementation of designated instructional 
materials. As can be seen in Table 3, 78 percent of the sessions included attention to issues of 
classroom pedagogy or use of the designated instructional materials, and 36 percent of the 
sessions included a focus on teacher content knowledge. 



Table 3 



Primary Intended Purposes of LSC Professional Development Sessions 







Percent of Sessions' 




All 


Science 


Mathematics 




Sessions 


K-8 


6-12 


K-8 


Explicit attention to classroom pedagogy/designated instructional 
materials 


78 


76 


69 


83 


Learning how to use specific instructional materials in the classroom 


42 


45 


42 


39 


Understanding student thinking/leaming about mathematics/science 
content 


35 


24 


31 


49 


Learning pedagogical/classroom management strategies 


32 


26 


34 


36 


Creating a vision of effective mathematics/science instruction 


25 


27 


21 


24 


Considering issues of scope and sequence (e.g., K-12 curricular 
frameworks) 


11 


8 


8 


16 


Designing or scoring student assessments 


9 


8 


8 


10 


Considering issues of access, equity, and diversity 


4 


0 


3 


8 


Learning how to use technology in the classroom 


4 


3 


13 


1 


Increasing mathematics/science content knowledge of participants 


36 


40 


47 


30 


Explicit attention to strategies/issues/roles of teacher leaders, 
principals, or others in leadership positions 


11 


13 


9 


11 


Other major purposes 


32 


20 


38 


41 


Promoting/exploring reflective practice 


19 


15 


20 


22 


Building professional networks among educators 


12 


7 


24 


15 


Orientation to the project 


2 


1 


2 


4 


Developing the capacity of participants to use technology 


6 


2 


21 


4 


Assessing participants’ knowledge/skills 


3 


1 


3 


6 


Involving administrators and/or other school/district personnel in the 
reform process 


3 


2 


0 


4 



Percents add to more than 100 as sessions could have multiple purposes. 
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Content Focus of Professional Development Sessions 

When sessions focused on one or more disciplinary content areas, evaluators were asked to 
categorize that content. In K-8 science projects, evaluators reported that slightly more than one- 
third of the sessions with a disciplinary content focus dealt with physical science concepts (38 
percent). Other content areas representing at least one-quarter of the sessions were life science 
(33 percent) and earth and space sciences (26 percent). Few of the observed K-8 science 
sessions dealt with “science as a way of knowing,” data collection or analysis, or engineering 
and design principles. (See Figure 5.) 



Content Areas of K-8 Science 
Professional Development Sessions^ 



Physical Science 



Life Science 

Earth and Space 
Sciences 

Science as a Way of 
Knowing 

Data collection and 
analysis 

Engineering and Design 
Principles 




Percentages add to more than 
100% because sessions could 
include more than one content area. 



Percent of Sessions 



Only sessions that focused on disciplinary content were included in these analyses. 



Figure 5 
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Projects targeting grade 6-12 science emphasized physical science in 52 percent of their sessions. 
(See Figure 6.) Other topics that were emphasized in more than 10 percent of the sessions were 
earth/space sciences (41 percent), life science (20 percent), and “science as a way of knowing” 
(14 percent). 



Content Areas of 6-12 Science 
Professional Development Sessions^ 




Percent of Sessions 

Percentages add to more than 
100% because sessions could 

include more than one content area. + Only sessions that focused on disciplinary content were included in these analyses. 



Figure 6 
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In projects targeting K-8 mathematics (see Figure 7), heavily emphasized topics included 
numeration and number theory (37 percent of the sessions that dealt with disciplinary content), 
computation (31 percent), and patterns and relationships (24 percent). Fewer than 20 percent of 
the disciplinary content sessions focused on measurement, geometry and spatial sense, data 
collection and analysis, estimation, or probability, and fewer than 10 percent addressed concepts 
in “mathematics as a way of knowing,” algebra, pre-algebra, functions, history of mathematics, 
or statistics. 




Session Activities 

The typical professional development session observed as part of the LSC core evaluation 
included several different types of activities. As can be seen in Table 4, most sessions included 
discussions or seminars (76 percent) and about half engaged participants in problem-solving or 
investigation. Thirty-eight percent of the sessions included formal presentations, usually by 
project staff as opposed to participants. Very few of the observed sessions involved participants 
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in reading (2 percent) or writing (3 percent) about disciplinary content, pedagogy or reform 
issues. 



Table 4 



Major Activities of LSC Professional Development Sessions 





Percent of Sessions 




All 


Science 


Mathematics 




Sessions 


K-8 


6-12 


K-8 


Engaged in discussions/seminars 


76 


72 


74 


83 


Whole group led by facilitator 


62 


63 


51 


68 


Small groups/pairs 


13 


10 


5 


19 


Whole group led by participants 


22 


17 


38 


19 


Engaged in problem-solving/investigation 


51 


58 


56 


43 


Listened to a formal presentation 


38 


42 


54 


32 


By presenter/facilitator 


37 


41 


51 


30 


By participants 


2 


2 


3 


2 


Read about disciplinary content, pedagogy, or reform issues 


2 


2 


5 


2 


Wrote about disciplinary content, pedagogy, or reform issues 


3 


4 


8 


0 



Percents add to more than 100 as sessions could include multiple activities. 



C. Quality of LSC Professional Development Sessions 

As part of assessing the quality of professional development sessions, evaluators were asked to 
rate a number of components for each session they observed, including the: 

• Design of the session; 

• Implementation of the professional development activities; 

• Quality of the disciplinary, pedagogical, and/or leadership content; and 

• Culture of the session. 

For each component area, observers first rated a series of individual indicators of best practice in 
professional development for standards-based mathematics/science education. These indicators 
were rated on a scale ranging from 1, “not at all” to 5, “to a great extent” to document the extent 
to which that feature characterized the observed professional development session. 

Considering those indicators, observers then assessed the overall quality of each component area. 
The lowest rating for component areas (Level 1) indicated that the session was not at all reflective 
of best practice. The highest rating (Level 5) indicated that the particular component of the 
session was extremely reflective of best practices for standards-based mathematics and science 
education.^* Evaluators’ ratings of the component areas are presented in the following sections. 



" Copies of the Professional Development Observation Protocol may be found in the Data Collection Manual 
section of the HRI web site; http://www.horizon-research.com/LSC. 
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Design of Professional Development Sessions 

As noted above, observers assessed the design of professional development sessions by rating a 
series of individual indicators based on current understandings of best practice. Several of these 
indicators received high ratings (4 or 5 on a five -point scale) in many of the observed sessions. 
Those indicators that were often highly rated included: 



• The extent to which the session reflected careful planning and organization (78 
percent); 

• The extent to which the session design provided opportunities for teachers to consider 
classroom applications of resources, strategies, and techniques (78 percent); and 

• The extent to which the session encouraged a collaborative approach to learning (75 
percent). 

Fewer sessions were rated highly on: 



• The extent to which the session provided adequate time and structure for “sense- 
making” (61 percent); and 

• The extent to which the session provided adequate time and structure for wrap-up (58 
percent). 

Overall observers found that the designs of the majority of the professional development sessions 
were generally reflective of best practice. As shown in Figure 8, 68 percent of the professional 
development sessions received overall design ratings of 4 or 5. 
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Implementation of Professional Development Sessions 

Observers also assessed the quality of implementation of professional development sessions. 
Indicators frequently receiving high ratings included: 



• The extent to which formal presentation(s) included in the session were carried out 
effectively (80 percent); and 

• The extent to which the facilitator(s)’ background, experience, and/or expertise 
enhanced the quality of the session (77 percent). 

As has been the case in previous years, fewer LSC professional development sessions were rated 
highly on such indicators as: 

• The extent to which the session modeled effective assessment strategies (66 percent); 
and 

• The extent to which the facilitators modeled questioning strategies that are likely to 
enhance the development of conceptual understanding (56 percent). 

As shown in Figure 9, 66 percent of the sessions received overall ratings of 4 or 5 on their 
quality of implementation. 
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Culture of Professional Development Sessions 

The literature on effective staff development emphasizes the importance of establishing a 
professional development culture where teachers can explore content and pedagogy in a 
collegial, risk-free environment. As can be seen in Figure 10, 73 percent of the sessions received 
synthesis ratings of 4 or 5 in this area. Indicators that were very likely to receive high ratings 
included: 



• The extent to which there was a climate of respect for participants’ experiences, ideas, 
and contributions (84 percent); and 

• The extent to which active participation of all was encouraged and valued (84 percent). 

Fewer sessions were rated highly on: 

• The extent to which participants were encouraged to generate ideas, questions, 
conjectures, and propositions (66 percent); 

• The extent to which intellectual rigor, constructive criticism, and the challenging of 
ideas were evident (66 percent); and 

• The extent to which participants demonstrated a willingness to share ideas and take 
intellectual risks (50 percent). 
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Disciplinary and Pedagogical Content of Professional Development Sessions 

Evaluators were asked to rate either the quality of the disciplinary content of the observed 
session, its pedagogical content, or both, depending on the focus of the session. Disciplinary 
content was rated in 201 of the 283 sessions, with 64 percent of these sessions receiving overall 
ratings of 4 or 5 in this area. (See Figure 11.) Indicators for disciplinary content that typically 
received high ratings included: 

• The extent to which the facilitators displayed an understanding of mathematics/science 
content (81 percent); and 

• The appropriateness of the disciplinary content for the purposes of the session and the 
backgrounds of the participants (80 percent). 

Fewer sessions received high ratings on: 

• The extent to which “sense-making” of mathematics/science content was appropriate 
for the purposes of the session (62 percent); and 

• The extent to which mathematics/science was portrayed as a dynamic body of 
knowledge continually enriched by conjecture, investigation, analysis, and/or 
proof/justification (58 percent). 




Horizon Research, Inc. 



19 



August 2006 





Observers rated 238 of the 283 observed professional development sessions on the quality of 
their pedagogical content. As can be seen in Figure 12, 64 percent of those professional 
development sessions received ratings of 4 or 5 for overall pedagogical content. 



Within the area of pedagogical content, sessions tended to be rated highly for: 

• The extent to which the depth and breadth of attention to instructional materials 
intended for classroom use were appropriate for the purposes of the session and 
participants’ needs (79 percent); and 

• The extent to which the facilitators displayed an understanding of pedagogical 
concepts (76 percent). 

Similarly to previous years, fewer sessions received high ratings for: 

• The extent to which depth and breadth of attention to student thinking/learning were 
appropriate for the purposes of the session and participants’ needs (64 percent); and 

• The extent to which “sense-making” about classroom practice was appropriate for the 
purpose of the session and the needs of adult learners (64 percent). 




Overall Assessment of Observed Professional Development Sessions 

In addition to rating the quality of individual components of the professional development 
session, observers were asked to assess the overall quality of each session. First, they considered 
the likely impact of the session on participants’ capacity for exemplary mathematics/science 
instruction, or the likely impact on leadership capacity when leadership development was a focus 
of the session instruction. They then assigned a “capsule rating” to characterize the overall 



Horizon Research, Inc. 



20 



August 2006 





quality of the professional development session. Ratings on a five-point scale ranged from 
“ineffective professional development” (Level 1) to “exemplary professional development” 
(Level 5). 

Impact on Participants’ Capacity for Exemplary Mathematics/Science Instruction 
Observers rated the likely impact of each session on teachers’ capacity for exemplary 
mathematics/science instruction. According to these observers, LSC professional development 
sessions often had a positive effect on participants’ abilities to network with other teachers about 
instruction (73 percent), use the designated instructional materials to develop students’ 
conceptual understanding (73 percent), and plan/provide high quality mathematics/science 
classroom instruction (73 percent). Sessions were also judged likely to have a positive effect on 
teachers’ self-confidence as mathematics/science instructors (69 percent), ability to identify and 
understand important ideas of mathematics/science (66 percent), understanding of how students 
learn (65 percent), and understanding of mathematics/science as a dynamic body of knowledge 
generated and enriched by investigation (58 percent). 

Quality of Leadership Development Sessions 

Many LSC projects incorporate the use of teacher leaders in their professional development 
strategies. When evaluators observed professional development sessions that focused on the 
preparation of teacher leaders, as did 32 of the 283 observed sessions, they were asked to rate a 
number of applicable key indicators in the area of leadership content. As can be seen in Figure 
13, 67 percent of the sessions focusing on leadership content received a high synthesis rating (4 
or 5) in this area. 
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When asked about the likely impact of the sessions on participants’ leadership capacity, each of 
the following areas was rated as likely to have a positive impact in 70 percent or more of the 
sessions: leaders’ knowledge and understanding of effective classroom practice; leaders’ 
understanding of important strategies for reform of mathematics/science education; leaders’ 
knowledge and understanding of mathematics/science; leaders’ ability to convey to others a 
vision of effective mathematics/science classrooms; leaders’ ability to plan/implement 
exemplary professional development; and professional networking among participants with 
regard to leadership roles. 

Capsule Ratings of Observed Professional Development Sessions 

As would be expected given the high ratings assigned by evaluators for the various components, 
overall ratings for individual professional development sessions were quite favorable. Only 2 
percent of observed LSC sessions were rated as ineffective professional development (Level 1), 
and 7 percent were rated at Level 2, having quite limited likelihood of helping participants 
implement exemplary mathematics/science instruction or be leaders in reform. Overall, 60 
percent of the observed professional development sessions received ratings of 4 or 5, indicating 
that those sessions were skillfully facilitated, engaging participants in purposeful work that 
would likely lead to enhanced capacity to implement exemplary instruction. (See Figure 14.) 
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D. Teacher Perceptions of the Overall Quality of LSC Professional 
Development Programs 

As part of the core evaluation, each year a sample of teachers is asked about the overall quality 
of the LSC professional development. A total of 221 teachers who had participated in 20 hours 
or more of LSC professional development were interviewed by project evaluators during the 
three years covered in this report. In addition, 3,998 teachers who had participated in LSC 
professional development answered survey questions about the quality of those experiences. 

Teachers who indicated they had participated in LSC professional development were asked to 
respond to a series of statements about those experiences. Table 5 shows that roughly one-third 
of the teachers who had participated in the LSC indicated that they were given considerable time 
to work with other teachers and to reflect on how to apply what they were learning to their 
classrooms. Forty-six percent of teachers indicated that they received considerable support for 
implementation. 



Table 5 



Teachers Agreeing^ to Statements about LSC Professional Development 





Percent of Teachers 




All 


Science 


Mathematics 




Teachers 


K-8 


6-12 


K-8 


I receive support as I try to implement what I’ve learned. 


46 


40 


43 


49 


I am encouraged to develop an individual professional development plan 
to address my needs and interests related to mathematics/science 
education. 


37 


27 


42 


41 


I am given time to work with other teachers as part of my professional 
development. 


34 


27 


32 


39 


I am given time to reflect on what I’ve learned and how to apply it to the 
classroom. 


32 


26 


28 


37 


I am involved in planning my mathematics/science-related professional 
development. 


30 


22 


39 


28 



^ Includes teachers indicating 4 or 5 on a five-point scale ranging from 1 “not at all” to 5 “to a great extent.” 



Table 6 shows teacher ratings of LSC professional development programs overall, with 6 percent 
of teachers rating the professional development programs “poor” or “very poor,” 55 percent 
“fair” or “good,” and 39 percent “very good” or “excellent.” 



Table 6 

Teacher Ratings of LSC Professional Development Programs Overall 





Percent of Teachers 


All Teachers 


Science 


Mathematics 


K-8 


6-12 


K-8 


Very Poor 


1 


1 


1 


1 


Poor 


5 


4 


4 


5 


Fair 


22 


21 


22 


23 


Good 


33 


34 


29 


33 


Very Good 


28 


28 


29 


27 


Excellent 


11 


11 


16 


10 
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Figure 15 shows the pereentage of non-lead teaehers in eaeh of the subjeet and grade-range 
combinations who rated LSC professional development “excellent” or “very good” in the 
questionnaires, analyzed by level of treatment. Note that the greater the level of participation, 
the higher the ratings. Similarly, Figure 16 shows the results on a composite variable on quality 
of the LSC professional development created from teachers’ responses to several items on the 
questionnaire.^^ For all three subject/grade-range groups, a larger number of hours of 
participation in LSC professional development is associated with higher composite scores. It is 
not possible to tell from these data whether teachers appreciate the professional development to a 
greater extent after having experienced more of it, or if the teachers who consider the 
professional development of high quality are more likely to participate in multiple professional 
development activities. 



Teachers Rating LSC Professional Development Excellent or Very Good, 

by Extent of Participation 




K-8 Mathematics 



U) 


100 -1 


d) 

£ 

o 

ra 


80 - 


d) 

1- 


60 - 


o 

c 

d) 


40 - 


o 

d) 

Q. 


20 - 



0 



1-39 40-99 100+ 

Hours of Professional Development 

Figure 15 



See Data Analysis in Section I for a description of how composite scores were calculated. 
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Composite: Teacher Perceptions of Quaiity of LSC Professionai Deveiopment, 

by Extent of Participation 






Figure 16 



Teacher interviews yielded similar findings. Evaluators asked a random sample of teachers who 
had participated in at least 20 hours of LSC professional development to talk about their 
experiences in the program and used these responses to characterize each teacher’s opinions on a 
five-point scale from very negative to very positive. Overall, 57 percent of teachers who had 
participated in LSC professional development had highly positive opinions of the LSC program 
and less than one percent had highly negative opinions. 

When asked about the impact of the LSC, about two-thirds of teachers talked about how the LSC 
had enabled them to change their classroom practice, specifically their instructional strategies, 
and half spoke about how the LSC had made them better prepared for mathematics/science 
teaching. Interestingly, more teachers cited improvements in relation to their instructional 
strategies than cited improvements related to content, either their own content knowledge or the 
content of their classroom instruction. Typical comments concerning changed classroom 
practice and increased preparedness follow. 
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Changed Classroom Practice 

I disliked science and felt I wasn ’t good at it. Having the classes [professional 
development] and being able to do it hands-on alleviated the stress. It strengthened my 
teaching and the science notebooking is something I use across other subjects now. I was 
burned out on texts, videos, and taking tests; I was turned off by that. I love the kits; it 
makes it fun for me and the students. (K-8 Science Teacher) 

I implemented the journaling. I have the students use their science notebook for every 
class kit. They have to show everything involved in the assignment like the procedures, 
set-up, etc. I have them do this in such detail so they can take their journal home every 
night and the parents know exactly what their child did in science for the day. (K-8 
Science Teacher) 

I do more inquiry. I have changed my focus from my being at center stage and now I 
redirect more responsibility on the shoulders of my students. I no longer teach the way I 
was trained. I lay the foundation, but I use less formal lecture and more questions. . .1 ask 
a lot of questions. (6-12 Science Teacher) 

It has provided me with the resources needed to conduct meaningful experiments. 

Instead of spending so much time locating items for experiments, lam able to spend time 
in preparation and planning. I can now focus more time on my students and the actual 
lesson. (6-12 Science Teacher) 

I think it has made me a better teacher. I am more familiar with the program and this 
helps me get more out of the students — using inquiry-based concepts — and get it to work 
best for the students. I am learning how to do the program and see how to break it down 
and how to focus. It has been really beneficial. I am more comfortable with math and 
definitely the hands-on approach. (K-8 Mathematics Teacher) 

I see myself as a facilitator. ..I do not have to be teaching all the time. They [ students ] do 
the work and the thinking. And the students like that too. They are not threatened or 
anything. They work individually and in groups. I did not do this as much before 
[project name] professional development. (K-8 Mathematics Teacher) 

It helped me grow professionally: new teaching ideas for lessons, making them more 
interesting, being able to talk with colleagues about what worked and didn ’t work, 
further developing those lessons to keep improving them. (6-12 Mathematics Teacher) 

Increased Preparedness 

It has made me more confident in my ability to teach science. And having the kits ready 
to use is wonderful. I really feel like I can do a good job of teaching science now. (K-8 
Science Teacher) 

Well, my understanding of content definitely improved and I felt much more confident 
about the material I was working with. (6-12 Science Teacher) 



Horizon Research, Inc. 



26 



August 2006 




Ifeel that it has benefited my teaching to a great extent. I felt 1 was a fairly good teacher 
to start with, but learning to use the [instructional] materials opened up a whole new 
world. It really helped me become an excellent teacher. I can see how things fit 
together. (K-8 Mathematics Teacher) 

When asked about the “most helpful” aspects of the LSC, 43 percent of the teachers identified 
the high quality of professional development, 40 percent of teachers mentioned getting materials 
needed for instruction, and 34 percent cited the opportunity to deepen their knowledge. About 
one-fifth of the teachers talked about the opportunities to deepen their knowledge of pedagogy 
and how to use the designated instructional materials. Science teachers were more likely than 
mathematics teachers to talk about the utility of the LSC in helping them understand content and 
providing them with materials needed for instruction. 

Teachers were also asked whether school and district policies and practices would help them 
maintain or expand the changes they have made in their teaching. The selection and provision of 
instructional materials, and the availability of high quality professional development were each 
mentioned by about one-third of the teachers as factors that would facilitate on-going reform. 
Science teachers were also likely to mention the more readily available equipment and supplies 
as a practice that would sustain the LSCs’ efforts. Typical comments included: 

The school and district continue to furnish our equipment and supplies. Now, we ’re all 
using the same materials and there in continuity between the grades and staff. (K-8 
Science Teacher) 

The professional development and training and the equipment and supplies provided are 
the things, I believe, helped me the most in adopting this program in our district. (6-12 
Science Teacher) 

Our superintendent sent out a letter to all teachers saying that [name of instructional 
materials] is here to stay and it’s expected to be taught in the classroom. (K-8 
Mathematics Teacher) 

All of the materials I would ever need in math have been ordered and I have received 
them. I’ve never been denied something that I needed. (6-12 Mathematics Teacher) 

When asked about needs for additional help in improving instruction, many teachers indicated no 
additional support was necessary or were unable to identify specific needs. When teachers did 
specify a need, they typically requested “more” of what they were already getting: more readily 
available materials or supplies, more professional development in pedagogy and in the use of the 
designated materials, more professional development in mathematics/science content, and more 
time for networking with other teachers. Typical comments included: 

Now that I teach first grade, I would like to start my 100 hours over again. . .1 would like 
to focus on what I am actually teaching. (K-8 Science Teacher) 

7 need more classes that help me learn about the content. (6-12 Science Teacher) 
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1 would like to see it [professional development] throughout the semester. You know, try 
it, go back, look at results, go back and follow-up, and then come back together and see, 
“Are these the results we wanted?” (K-8 Mathematics Teacher) 

Teachers also indicated that high-stakes testing had limited the impacts of the LSCs, or would 
make the reforms difficult to sustain. Nineteen percent of interviewed mathematics teachers 
indicated that student grading or testing policies have hindered progress, with the lack of 
alignment of assessments with the designated instructional materials and the number of topics 
they need to cover to prepare students for the test being the main concerns. Science instruction 
has also been affected by high-stakes testing; 28 percent of interviewed science teachers 
indicated that science instruction was suffering because of the priority given to other disciplines. 
Typical comments included: 

As a sixth grade teacher, I had to get my students ready to take the [state test]. That 
means that I can ’t take as much time for science as I would like to. (K-8 Science 
Teacher) 

School reform focused on reading and mathematics. We used to have an emphasis on 
science, science festival, family science, and kids would do hands-on science 
experiments. Now, there is so much emphasis on test scores. To be honest, I focus on 
math a lot. (6-12 Science Teacher) 

We don’t always go deep with the curriculum because of pressures from the tests and 
covering the standards. (K-8 Mathematics Teacher) 



E. Evaluator Ratings of the Quality of ESC Professional Development Programs 

Based on the results of their observations, as well as feedback from participating teachers, 
evaluators rated the overall quality of the LSC professional development in a number of areas, 
including preparing project staff to carry out their roles in providing professional development to 
targeted teachers; the quality of the professional development culture; the project’s overall 
treatment of disciplinary content; instructional materials and pedagogy; and the nature and extent 
of support provided to teachers during implementation. 

Preparedness of Professional Development Providers 

As can be seen in Figure 17, overall, 86 percent of LSC projects received high ratings (4 or 5 on 
a five-point scale ranging from 1 “inhibited effective professional development” to 5 “facilitated 
effective professional development”) for the quality of their efforts in preparing professional 
development providers. 
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Evaluator Ratings: 
Preparedness of Professional 
Development Providers 




Rating 

Figure 1 7 



Professional Development Culture 

Using all of the information available to them, including teacher comments and their own 
observations, evaluators rated the overall success of each project in creating a climate conducive 
to teacher learning. Overall, 91 percent of projects received ratings of 4 or 5 in this area, 
suggesting that the professional development culture is a major strength of these projects. (See 
Figure 18.) 



Evaluator Ratings: 
Professional Development Culture 




Rating 

Figure 18 
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Treatment of Disciplinary Content 

When they prepared their annual reports, evaluators considered the data they had from 
observations, interviews, and questionnaires and came up with an overall rating of the quality of 
the project’s treatment of disciplinary content. As can be seen in Figure 19, only 58 percent of 
projects received high ratings (4 or 5 on a five-point scale ranging from 1 “poor” to 5 
“excellent”) in this area. 



Evaluator Ratings: 

Quality of Treatment of Disciplinary Content 




Rating 

Figure 19 



Treatment of Instructional Materials and Pedagogy 

In addition to describing the quality of the project’s treatment of the designated instructional 
materials and pedagogy, evaluators were asked to provide overall ratings in this area. As can be 
seen in Figure 20, 86 percent of projects received ratings of 4 or 5 in this area, markedly higher 
than the 58 percent in developing disciplinary content. 
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Evaluator Ratings: 

Quality of Treatment of Instructional 
Materials and Pedagogy 




Rating 

Figure 20 



Support for Teachers During Implementation 

Based on interview, observation, and questionnaire data, evaluators provided an overall rating of 
the quality of the support provided to teachers as they implemented the instructional materials in 
their classrooms. As can be seen in Figure 21, overall, 72 percent of projects received high 
ratings in this area; 9 percent received a rating below 3 on a five-point scale. 



Evaluator Ratings: 

Support for Teachers During Implementation 




Rating 



Figure 21 
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Continuum Ratings 

At the close of the data collection year, evaluators were asked to use all of the information 
available to them to place the project on a continuum, from predominance of ineffective 
professional development, through various stages of improvement, to a system of predominantly 
well-designed professional development. As can be seen in Table 7, nearly three-quarters of the 
LSC projects were rated as having an emerging infrastructure or a predominance of well- 
designed professional development; none were rated at the lowest level. 



Table 7 

Continuum Ratings for Quality of LSC Professional Development 





Percent of Projects 


Level 1: Predominance of Ineffective Professional Development 


0 


Level 2: Exploring Quality Professional Development 


5 


Level 3: Transitioning to Quality Professional Development 


23 


Level 4: Emerging Infrastructure of Well-Designed Professional Development 


41 


Level 5: Predominance of Well-Designed Professional Development 


32 


Mean Continuum Rating Level 


4.0 
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III. Impact of the LSC on Teacher 
Preparedness, Attitudes, and Beliefs 



The theory of action underlying the Local Systemic Change initiative argues that providing 
teachers with well-designed opportunities to appreciate standards-based reform and deepen their 
content and pedagogical knowledge in the context of high-quality instructional materials will 
result in better prepared teachers. When these teachers are also given support in using these 
instructional materials, the theory predicts, they will be both inclined to change their teaching in 
ways advocated by national standards, and have the capability of doing so. Improved 
instruction, in turn, will lead to higher student achievement. 

Participating in LSC professional development impacted teachers’ beliefs about mathematics/ 
science education in a variety of ways, prompting them to re-evaluate their own practice as well 

1 o 

as their perceptions about mathematics and science teaching. The reflection time built into 
high-quality professional development sessions gave teachers the opportunity to process what 
they had learned about content and pedagogy, and to examine their evolving beliefs about 
teaching and learning. Still, many teachers continue to feel under-prepared in these areas. 

As can be seen in Figure 22, results on a composite of several items related to teachers’ attitudes 
toward standards-based teaching indicated that most teachers, regardless of treatment level, had 
positive attitudes. (Effect sizes, and the data used to calculate them, for all comparisons on the 
composites are included in the Appendix.'”^) There were no significant differences between 
highly treated teachers and untreated teachers on this composite for any of the subject/grade- 
range combinations. 



As mentioned previously, teacher leaders were excluded from all analyses examining the impact of the LSC on 
teachers. 

The effect size for composites, Cohen’s d, is calculated as the difference between the “0 hours” and ‘TOO or more 
hours” group means, divided by the pooled standard deviation. Following standard conventions, effect sizes of 0.2 
are considered small effects, 0.5 medium effects, and 0.8 large effects (Jacob Cohen, Statistical Power Analysis for 
the Behavior Sciences, Hillsdale, NJ: Lawrence Erlbaum Associates, 1988). 
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Composite: Teacher Attitudes Toward Teaching, 
by Extent of Participation in LSC Professionai Deveiopment 




Figure 22 



Participating in LSC professional development appears to have had a substantial impact on 
elementary teaehers’ feelings of preparedness to teach science. As ean be seen in Figure 23, 88 
percent of K-8 science teaehers who had participated in at least 100 hours of LSC professional 
development indicated they were at least fairly well prepared to teach science, compared to 72 
percent of those who had not yet participated in LSC professional development (an effect size^^ 
of 0.41). 



When comparing percents, the effect size, h, is calculated using the difference between the arcsine transformation 
of the percents of the “0 hours” and “100 or more hours” groups. Specifically, h = (pi - 92 , where 9 = 2*arcsin(P''^), 
and P is the proportion of cases in a category. 
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K-8 Science Teachers Feeling 
at Least Fairly Well-Prepared 
to Teach Science 




Hours of Professional Development 



Figure 23 



In mathematics, the vast majority (87 percent) of teachers in K-8 projects who had not yet 
received LSC professional development indicated they were at least fairly well prepared to teach 
mathematics, so the comparison was made for very well prepared. As can be seen in Figure 24, 
71 percent of K-8 mathematics teachers who had participated in at least 100 hours of LSC 
professional development indicated they were very well prepared to teach mathematics, 
compared to 50 percent of those who had not yet participated in LSC professional development 
(an effect size of 0.43). 



K-8 Mathematics Teachers 
Feeling Very Well-Prepared 
to Teach Mathematics 




Hours of Professional Development 



Figure 24 



Horizon Research, Inc. 



35 



August 2006 




A similar pattern can be seen when teachers were asked about their preparedness to teach 
specific science and mathematics topics. In K-8 science, there were significant differences 
between untreated and highly-treated teachers on 3 of the 1 1 topics listed, with effect sizes 
ranging from 0.20 to 0.37. (See Table 8.) 



Table 8 



K-8 Science Teachers Feeling at Least Fairly Well-Prepared^ to Teach 
Each Topic, by Extent of Participation in LSC Professional Development 





Percent of Teachers 


Effect Size 

(h) 


0 

Hours 
(N = 417) 


1-39 
Hours 
(N = 576) 


40-99 
Hours 
(N = 361) 


100 or More 
Hours 
(N = 582) 


The human body 


67 


65 


69 


68 


0.02 


Ecology 


55 


58 


60 


65 


0.20* 


Rocks and soils 


51 


62 


63 


69 


0.37* 


Astronomy 


46 


48 


49 


51 


0.10 


Processes of change over time 


42 


45 


47 


47 


0.10 


Mixtures and solutions 


44 


53 


59 


56 


0.24* 


Electricity 


43 


48 


50 


52 


0.18 


Sound 


48 


49 


57 


55 


0.14 


Eorces and motion 


54 


55 


61 


61 


0.14 


Machines 


47 


47 


52 


50 


0.06 


Engineering and design principles 


28 


29 


33 


30 


0.04 



Includes teachers indicating 3 or 4 on a four-point scale ranging from 1 “not adequately prepared” to 4 “very well prepared.” 



* p < 0.05; (proportion of teachers feeling at least fairly well-prepared in the 100 or more hours group significantly different 
than in the 0 hours group, RS3-adjusted chi-square, WesVar 4.2). 



Participation in LSC professional development also appears to have had an impact on secondary 
science teachers’ perceptions of their content preparedness, with significant differences between 
untreated and highly-treated teachers in 9 of the 22 topics, including all four of the chemistry 
topics listed on the questionnaire. (See Table 9.) 
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Table 9 



6-12 Science Teachers Feeling at Least Fairly Well-Prepared^ to 
Teach Each Topic, by Extent of Participation in LSC Professional Development 







Percent of Teachers 








0 

Hours 

(N = 72) 


1-39 

Hours 

(N = 152) 


40-99 
Hours 
(N = 142) 


100 or More 
Hours 
(N = 166) 


Effect Size 
(h) 


Earth Science 












The solar system and the universe 


53 


69 


65 


73 


0.42 


Earth’s features and physical processes 


51 


81 


72 


85 


0.76* 


Climate and weather 


54 


67 


65 


67 


0.27 


Biology 












Animal behavior 


58 


63 


68 


71 


0.27 


Genetics and evolution 


63 


60 


65 


72 


0.19 


Interactions of living things/ecology 


70 


76 


82 


87 


0.42 


Plant biology 


57 


63 


67 


69 


0.25 


Structure and functions of human 












systems 


72 


71 


77 


79 


0.16 


Chemistry 












Properties and states of matter 


56 


68 


76 


85 


0.66* 


Structure of matter and chemical 












bonding 


46 


55 


62 


67 


0.43* 


Chemical reactions 


44 


55 


66 


68 


0.49* 


Energy and chemical change 


44 


58 


67 


73 


0.60* 


Physics 












Forces and motion 


53 


63 


66 


78 


0.53* 


Electricity and magnetism 


41 


48 


49 


60 


0.38 


Energy 


50 


59 


64 


75 


0.52* 


Light and sound 


49 


54 


59 


67 


0.37 


Modem physics (e.g., special relativity) 


29 


28 


28 


31 


0.04 


Environmental and Resource Issues 












Pollution, acid rain, global warming 


63 


68 


73 


78 


0.33* 


Population, food supply and production 


66 


63 


75 


83 


0.40 


Scientific Methods and Inquiry Skills 

Describing, graphing, and interpreting 












data 


82 


89 


92 


94 


0.38 


Formulating hypotheses, drawing 












conclusions, making generalizations 


74 


91 


91 


94 


0.58* 


Experimental design 


65 


80 


84 


86 


0.50* 



Includes teachers indicating 3 or 4 on a four-point scale ranging from 1 “not adequately prepared” to 4 “very well prepared.” 
* p < 0.05; (proportion of teachers feeling at least fairly well-prepared in the 100 or more hours group significantly different 



than in the 0 hours group, RS3-adjusted chi-square, WesVar 4.2). 
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In K-8 mathematics, teachers with 100 or more hours of LSC professional development were 
significantly more likely than untreated teachers to indicate that they were at least fairly well- 
prepared to teach 2 of the 1 1 topics listed. (See Table 10.) 



Table 10 

K-8 Mathematics Teachers Feeling at Least Fairly Well-Prepared^ to 
Teach Each Topic, by Extent of Participation in LSC Professional Development 





Percent of Teachers 


Effect Size 
(h) 


0 

Hours 
(N = 231) 


1-39 
Hours 
(N = 438) 


40-99 
Hours 
(N = 553) 


100 or More 
Hours 

(N = 554) 


Computation 


96 


97 


98 


99 


0.20 


Numeration and number theory 


92 


93 


95 


97 


0.23 


Estimation 


91 


93 


94 


95 


0.16 


Measurement 


91 


89 


94 


95 


0.16 


Pre-algebra 


75 


76 


81 


85 


0.25 


Algebra 


64 


67 


71 


74 


0.22 


Patterns and relationships 


89 


93 


94 


98 


0.39* 


Geometry and spatial sense 


78 


85 


86 


90 


0.33 


Data collection and analysis 


80 


85 


90 


92 


0.35* 


Probability 


75 


74 


79 


81 


0.15 


Technology in support of mathematics 


66 


65 


70 


66 


0.00 



Includes teachers indicating 3 or 4 on a four-point scale ranging from 1 “not adequately prepared” to 4 “very well 
prepared.” 



* p < 0.05; (proportion of teachers feeling at least fairly well-prepared in the 100 or more hours group significantly 
different than in the 0 hours group, RS3-adjusted chi-square, WesVar 4.2). 
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When the various topic areas were combined into a single composite score for content 
preparedness, a significant difference was found between teachers with no treatment and those 
with 100 or more hours of LSC professional development in 6-12 science (effect size of 0.62 
standard deviations). (See Figure 25.) No significant differences were found for K-8 
mathematics or science. 



Composite: Teacher Perceptions of Their Science/Mathematics Content Preparedness, 
by Extent of Participation in LSC Professionai Deveiopment 



K-8 Science 




6-12 Science 




Hours of Professional Development 



Hours of Professional Development 



K-8 Mathematics 




Hours of Professional Development 



Figure 25 
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Questionnaire data on other items provide support for the impact of the LSC on teacher self- 
confidence. For example, the larger the number of hours of LSC professional development, the 
more likely teachers were to indicate that they are well-informed about national 
mathematics/science standards. (See Figure 26.) 



Teachers Indicating They Are Well-Informed about National Standards, 
by Extent of Participation in LSC Professional Development 



K-8 Science 




6-12 Science 




Hours of Professional Development 



Hours of Professional Development 



K-8 Mathematics 




Hours of Professional Development 



Figure 26 
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Figure 27 shows the results on a composite of items about teacher preparedness to use a variety 
of instructional strategies in their mathematics/science instruction, including taking students’ 
prior understanding into account when planning curriculum and instruction, having students 
work in cooperative learning groups, and using informal questioning to assess student 
understanding. The ten percentage point difference between untreated and highly-treated 6-12 
science teachers constitutes a large effect (0.78 standard deviations). For K-8 mathematics, the 
four point difference between untreated and highly treated teachers represents a small effect 
(0.33 standard deviations). There was no significant difference on this composite for K-8 
science teachers. 







100 -1 


ro 

o 


(0 

c 


80 - 


1- 


o 


60 - 




Q. 


o 


0> 


c 

0> 

Q 


n 

<n 


40 - 


0> 

Q. 


o 

a. 


20 - 



Composite: Teacher Perceptions of Their Pedagogicai Preparedness, 
by Extent of Participation in LSC Professionai Deveiopment 



77 



K-8 Science 



79 



80 



80 



0 1-39 40-99 100+ 

Hours of Professional Development 







100 -1 


o 


U) 

c 


80 - 


1- 


o 


60 - 




Q. 


o 


O 


c 

o 

o 


'« 

ui 


40 - 


d) 

Q. 


O 

Q. 


20 - 



ra <3 
o .£ 
I- o 

c n 
o “ 

o> o 
a a 



100 

80 

60 

40 

20 

0 



6-12 Science 



71 



77 



79 



81 



0 1-39 40-99 100+ 

Hours of Professional Development 



K-8 Mathematics 

80 80 81 



0 1-39 40-99 100+ 

Hours of Professional Development 



Figure 27 
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IV. Impact of the LSC on Classroom Practice 



A. Introduction 

The core evaluation focuses a great deal of attention on the impact of the LSC projects on 
classroom instruction. Data come from several sources: classroom observations, teacher 
interviews, and teacher questionnaires. As was the case with impact on teachers, the impact of 
the LSC on classroom practice was assessed by comparing results for teachers with varying 
extents of participation in LSC professional development. 



B. Time Spent on Elementary Science Instruction 

Over the years, one of the impacts of the LSC has been increased attention to science instruction 
in the elementary grades. However, as can be seen in Figures 28-31, there are no significant 
differences in this area for the elementary science data included in this report. Based on teacher 
interview data, the increased attention to mathematics and literacy in the face of high-stakes 
accountability may help explain this finding. 



Average Number of Days of Instruction in 
Science in the Last Five School Days, 
by Extent of Participation in 
LSC Science Professional Development 
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Figure 28 
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Figure 29 




Figure 30 
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Elementary Classrooms Where Science Units 
Are Typically Four Weeks or More in Length, 
by Extent of Participation in 
LSC Science Professional Development 




Hours of Professional Development 

Figure 31 



C. Instructional Strategies 

One indication of the impact of LSC activities on classroom practice comes from composites 
created from questionnaire data. The investigative culture composite includes strategies used by 
teachers to facilitate exploration and investigation by students. It includes such practices as: 

• Arranging seating to facilitate student discussion; 

• Using open ended questions; 

• Requiring students to supply evidence to support their claims; and 

• Encouraging students to consider alternative explanations. 

There was a significant increase in scores on this composite with increasing participation in LSC 
activities for all three targeted subject/grade-level combinations. (See Figure 32.) The nine- 
point difference between untreated and highly treated teachers in 6-12 science represents a large 
effect size (0.79 standard deviations). The seven-point difference in K-8 mathematics represents 
a medium effect size (0.54 standard deviations), and the six-point difference in K-8 science 
represents a small effect size (0.43 standard deviations). 
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K-8 Mathematics 




Hours of Professional Development 



Figure 32 



The investigative practices composite is tied to what students actually do in the classroom. It 
includes such instructional strategies as having students: 

• Engage in hands-on mathematics/science activities; 

• Work on models or simulations; 

• Work on extended investigations; and 

• Write reflections in a notebook or journal. 

As shown in Figure 33, there was an increase in composite scores for both science and 
mathematics teachers with increasing participation in ESC activities. The differences between 
untreated and highly-treated teachers represent a medium effect size for 6-12 science (0.56 
standard deviations), and small effect sizes for K-8 science and mathematics (0.39 and 0.37 
standard deviations, respectively). 
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Figure 33 
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D. Quality of Observed Lessons 



Trained observers visited classrooms of teachers who had already participated in LSC 
professional development and others who had not yet participated and assessed the quality of the 
lessons using a variety of indicators. (See box.) 



Sample Indicators for Classroom Observations 

Design 

• The design of the lesson incorporated tasks, roles, and interactions consistent with investigative 
mathematics/science. 

• The design of the lesson reflected careful planning and organization. 

• The instructional strategies and activities used in this lesson reflected attention to students’ experience, 
preparedness, and/or learning styles. 

• The resources available in this lesson contributed to accomplishing the purposes of the instruction. 

• The design of the lesson encouraged a collaborative approach to learning. 

• Adequate time and structure were provided for “sense-making.” 

• Adequate time and structure were provided for wrap-up. 

Implementation 

• The instruction was consistent with the underlying approach of the instructional materials designated for 
use by the LSC. 

• The teacher’s classroom management style/strategies enhanced the quality of the lesson. 

• The pace of the lesson was appropriate for the developmental levels/needs of the students and the 
purposes of the lesson. 

• The teacher was able to “read” the students’ level of understanding and adjust instruction accordingly. 

• The teacher’s questioning strategies were likely to enhance the development of student conceptual 
understanding/problem solving (e.g., emphasized higher order questions, appropriately used “wait 
time,” identified prior conceptions and misconceptions). 

Mathematics/Science Content 

• The mathematics/science content was significant and worthwhile. 

• The mathematics/science content was appropriate for the developmental levels of the students in this 
class. 

• Students were intellectually engaged with important ideas relevant to the focus of the lesson. 

• Teacher-provided content information was accurate. 

• Appropriate connections were made to other areas of mathematics/science, to other disciplines, and/or 
to real-world contexts. 

Classroom Culture 

• Active participation of all was encouraged and valued. 

• There was a climate of respect for students’ ideas, questions, and contributions. 

• Interactions reflected collegial working relationships among students (e.g., students worked together, 
talked with each other about the lesson). 

• The climate of the lesson encouraged students to generate ideas, questions, conjectures, and/or 

propositions. 
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Observers then rated the quality of each lesson’s design and implementation, the science/ 
mathematics content, and the classroom culture. Lessons of treated teachers (those who had 
participated in 20 or more hours of LSC professional development) were not significantly 
different in the quality of their design, implementation, content, and classroom culture than 
lessons of teachers who had not yet participated (see Figure 34).*^’^^ 



Highly-Rated Lessons in Each Area, 
by T reatment 



□ Untreated Teachers 

□ Treated Teachers 




Figure 34 



While questionnaire results were typically presented for four levels of participation in LSC professional 
development (0, 1-39, 40-99, and 100 or more hours), the considerably smaller number of classroom observations 
prevented that extent of disaggregation. 

Although 366 classroom observations were submitted during the time frame covered by this report, only 257 
were included in these analyses. Data from the 47 lessons taught by teacher leaders were excluded as were the 48 
lessons taught by teachers that did not return a completed teacher questionnaire (by which the treatment information 
is gathered). An additional 14 lessons were excluded as the observed teachers completed a questionnaire, but did 
not respond to the item asking about the extent of their participation in LSC professional development. Of the 257 
lessons analyzed, 59 were taught by untreated teachers and 198 were taught by treated teachers. 
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Classroom observers also considered the potential for student impact as they observed lessons. 
Areas of likely student impact are compared for treated and untreated teachers in Figure 35. 
Lessons of treated and untreated teachers were not statistically different in any of these areas. 



100 n 



Lessons Judged Likely to Have a 
Positive Impact on Students, 
by T reatment 



n Untreated Teachers 
□ Treated Teachers 



80 A 




View Understand Capacity to do Ability to appiy Self-confidence Interest in 

math/science as math/science own inquiries learning in math/science math/science 

dynamic concepts 



Figure 35 



In addition, each lesson was given an overall capsule rating. Overall quality ratings for lessons 
of treated and untreated teachers were not significantly different. (See Figure 36.) 
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Figure 36 



Since a specific goal of the LSC program is to increase the use of exemplary instructional 
materials, classroom observers were asked to note whether or not these materials were being 
used and to comment on the quality of their use. As can be seen in Figure 37, the majority of 
observed lessons were based on the designated instructional materials, regardless of whether the 
teacher had participated in LSC professional development. The widespread use of the designated 
instructional materials may be responsible for the lack of difference in lesson quality between 
treated and untreated teachers. 




Figure 37 
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However, lesson quality is related to teachers’ use of the designated instructional materials, 
reinforcing the LSC program’s emphasis on having teachers implement the designated 
instructional materials as designed by their developers. As can be seen in Figure 38, 50 percent 
of the lessons that adhered closely to the materials were given high ratings (capsule ratings of 4 
or 5), significantly more than the 23 percent of lessons not using the designated instructional 
materials. 



Highly Rated Lessons, by Use of Designated 
Instructional Materials 




Not Using Using Designated Using Designated 

Designated Materiais with Low Materiais with High 
Materiais Fideiity Fideiity 



Figure 38 
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V. Conclusions 



Results from the Local Systemic Change core evaluation between 2003 and 2006 continue to 
show areas of both strength and weakness in the design and implementation of the professional 
development and the impact of those interventions on teachers and their teaching. 

Evaluators were asked to observe a representative sample of professional development sessions 
and rate each in relation to its particular purposes. Sessions were very likely to receive high 
ratings for providing a climate of respect, encouraging active participation, and fostering a 
collaborative approach to learning, as well as addressing appropriate mathematics/science 
content. The most salient weaknesses in sessions for classroom teachers were in questioning 
participants in ways likely to enhance their conceptual understanding, creating a culture that 
encouraged participants to share ideas and take intellectual risks, providing adequate time and 
structure for wrap-up, portraying mathematics/science as a dynamic body of knowledge, and the 
appropriateness of “sense-making” about classroom practice. 

Interestingly, the majority of the observed sessions were facilitated by district personnel, most 
often full- or part-time teacher leaders. Fewer than 1 in 6 sessions included scientists or 
mathematicians as professional development providers, and fewer than 2 in 5 had a major focus 
on increasing teacher content knowledge, raising the concern that the ESC professional 
development does not emphasize adequately the need to deepen teacher disciplinary content 
knowledge. Similarly, just over one-third of the observed sessions included a focus on helping 
teachers understand student thinking/learning about mathematics or science content, an area that 
is increasingly being identified as important in teacher development. 

While only 39 percent of the teachers rated the ESC professional development excellent or very 
good, the more hours of participation in ESC professional development, the higher the ratings of 
quality. In interviews, teachers indicated that having the opportunity to deepen their content and 
pedagogical knowledge, receiving materials needed for instruction, the high quality of ESC 
professional development, and the opportunities to collaborate with other teachers were 
particularly helpful aspects of the ESC. 

Questionnaire data collected from targeted teachers suggest that ESC professional development 
has had a significant impact on teachers’ perceptions of their pedagogical preparedness. Impact 
on teachers’ perceptions of content preparedness was mixed, with significant differences found 
only for 6-12 science projects. In addition, participants reported making greater use of strategies 
that facilitate student exploration and investigation by students than did teachers who had not 
participated in ESC professional development, such as using open ended questions and requiring 
students to supply evidence to support their claims. Participants were also more likely than other 
teachers to use reform-oriented teaching practices such as having students engage in hands-on 
activities, work on extended investigations, and write reflections in notebooks or journals. 

For the time period covered in this report, the quality of observed lessons was no different for 
teachers who had and had not participated in ESC professional development. This finding could 
indicate that ESC professional development was not as effective as desired. The data also 
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indicate that lessons that adhered closely to the designated materials were more than twice as 
likely to be rated highly than lessons not based on the designated materials. Thus, the impact of 
LSC professional development may be masked by the widespread use, in both treated and 
untreated teachers’ classrooms, of the designated instructional materials. 
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Appendix Tables 



Summary of the Impact of LSC Professional Development on 
Teacher Perceptions of Their Preparedness and on Their Teaching 



K-8 Science 



Questionnaire Composite 


Mean 


Pooled 

Standard Deviation 


Effect 
Size (d) 


0 Hours 
(N = 418) 


100 or More Hours 
(N = 583) 


Attitudes Toward Teaching 


86.68 


86.68 


8.95 


0.00 


Pedagogical Preparedness 


76.71 


79.93 


12.16 


0.26 


Content Preparedness 


61.27 


64.07 


17.16 


0.16 


Investigative Culture 


73.66 


79.92 


14.61 


0.43* 


Investigative Practices 


50.38 


55.89 


14.08 


0.39* 



* p < 0.05; (composite score for the 100 or more hours group significantly greater than for the 0 hours group, 1-tailed 



independent samples t-test - WesVar 4.2) 



6-12 Science 



Questionnaire Composite 


Mean 


Pooled 

Standard Deviation 


Effect 
Size (d) 


0 Hours 
(N = 73) 


100 or More Hours 
(N = 166) 


Attitudes Toward Teaching 


82.24 


86.60 


9.15 


0.48 


Pedagogical Preparedness 


70.56 


80.65 


12.92 


0.78* 


Content Preparedness 


73.43 


83.23 


15.78 


0.62* 


Investigative Culture 


71.63 


81.10 


11.99 


0.79* 


Investigative Practices 


48.41 


55.30 


12.34 


0.56* 



* p < 0.05; (composite score for the 100 or more hours group significantly greater than for the 0 hours group, 1-tailed 



independent samples t-test - WesVar 4.2) 



K-8 Mathematics 



Questionnaire Composite 


Mean 


Pooled 

Standard Deviation 


Effect 
Size (d) 


0 Hours 
(N = 231) 


100 or More Hours 
(N = 554) 


Attitudes Toward Teaching 


88.79 


89.23 


9.35 


0.05 


Pedagogical Preparedness 


79.97 


84.25 


12.92 


0.33* 


Content Preparedness 


76.09 


79.79 


15.98 


0.23 


Investigative Culture 


81.41 


88.27 


12.70 


0.54* 


Investigative Practices 


50.03 


56.22 


16.67 


0.37* 


Use of Calculators and Computers 


54.37 


57.08 


22.16 


0.12 



* p < 0.05; (composite score for the 100 or more hours group significantly greater than for the 0 hours group, 1-tailed 
independent samples t-test - WesVar 4.2) 
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Report Available on the Web 



This report is available on Horizon Research, Inc.’s web site: 
WWW .horizon-re search . com/LSC 



Look for: 

Local Systemic Change through Teacher Enhancement: 2003-06 Cross-Site Report 
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